Project - Bank Churn Prediction

Background:

Businesses like banks that provide service have to worry about the problem of 'Churn' i.e. customers leaving and joining another service provider. It is important to understand which aspects of the service influence a customer's decision in this regard. Management can concentrate efforts on the improvement of service, keeping in mind these priorities.

Objective:

To identify different segments in the existing customer based on their spending patterns as well as past interaction with the bank.

Dataset:

https://www.kaggle.com/barelydedicated/bank-customer-churn-modeling

Attributes:

• RowNumber: Row number.
• CustomerId: Unique identification key for different customers.
• Surname: Surname of the customer.
• Credit Score: Credit score is a measure of an individual's ability to pay back the borrowed amount. It is the numerical representation of their creditworthiness. A credit score is a 3-digit number that falls in the range of 300-900, 900 being the highest.
• Geography: The country to which the customer belongs.
• Gender: The gender of the customer.
• Age: Age of the customer.
• Tenure: The period of time a customer has been associated with the bank.
• Balance: The account balance (the amount of money deposited in the bank account) of the customer.
• NumOfProducts: How many accounts, bank account affiliated products the person has.
• HasCrCard: Does the customer have a credit card through the bank?
• IsActiveMember: Subjective, but for the concept.
• EstimatedSalary: Estimated salary of the customer.
• Exited: Did they leave the bank after all?

Import Necessary Libraries & Format Notebook

# Supress warnings
import warnings
warnings.filterwarnings('ignore')

# Import libraries
import numpy as np
import pandas as pd

import matplotlib.pyplot as plt 
import seaborn as sns
import pandas_profiling

from sklearn.preprocessing import StandardScaler

from sklearn.model_selection import train_test_split

import tensorflow as tf
from tensorflow.keras.layers import Input, Dense

from sklearn.metrics import roc_auc_score, accuracy_score

# Adjust display and formatting settings
from IPython.core.display import display, HTML
display(HTML("<style>.container { width:98% !important; }</style>"))

plt.style.use('ggplot')
plt.rcParams['figure.figsize'] = [30, 8]
pd.options.display.float_format = '{:,.3f}'.format
pd.set_option('display.max_columns', 500)           
pd.set_option('display.max_rows', 500)

# Check tensorflow version
tf.__version__

'2.3.1'

Exploratory Data Analysis

Initial Data Observations

# load the dataset from the CSV file
df = pd.read_csv('bank.csv')

# Visualize the data
df.head(10)

	RowNumber	CustomerId	Surname	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
0	1	15634602	Hargrave	619	France	Female	42	2	0.000	1	1	1	101,348.880	1
1	2	15647311	Hill	608	Spain	Female	41	1	83,807.860	1	0	1	112,542.580	0
2	3	15619304	Onio	502	France	Female	42	8	159,660.800	3	1	0	113,931.570	1
3	4	15701354	Boni	699	France	Female	39	1	0.000	2	0	0	93,826.630	0
4	5	15737888	Mitchell	850	Spain	Female	43	2	125,510.820	1	1	1	79,084.100	0
5	6	15574012	Chu	645	Spain	Male	44	8	113,755.780	2	1	0	149,756.710	1
6	7	15592531	Bartlett	822	France	Male	50	7	0.000	2	1	1	10,062.800	0
7	8	15656148	Obinna	376	Germany	Female	29	4	115,046.740	4	1	0	119,346.880	1
8	9	15792365	He	501	France	Male	44	4	142,051.070	2	0	1	74,940.500	0
9	10	15592389	H?	684	France	Male	27	2	134,603.880	1	1	1	71,725.730	0

# Shape of our dataset
df.shape

(10000, 14)

# Display data types and determine if any missing values
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000 entries, 0 to 9999
Data columns (total 14 columns):
 #   Column           Non-Null Count  Dtype  
---  ------           --------------  -----  
 0   RowNumber        10000 non-null  int64  
 1   CustomerId       10000 non-null  int64  
 2   Surname          10000 non-null  object 
 3   CreditScore      10000 non-null  int64  
 4   Geography        10000 non-null  object 
 5   Gender           10000 non-null  object 
 6   Age              10000 non-null  int64  
 7   Tenure           10000 non-null  int64  
 8   Balance          10000 non-null  float64
 9   NumOfProducts    10000 non-null  int64  
 10  HasCrCard        10000 non-null  int64  
 11  IsActiveMember   10000 non-null  int64  
 12  EstimatedSalary  10000 non-null  float64
 13  Exited           10000 non-null  int64  
dtypes: float64(2), int64(9), object(3)
memory usage: 1.1+ MB

# Display statistical metrics of our data
df.describe().transpose()

	count	mean	std	min	25%	50%	75%	max
RowNumber	10,000.000	5,000.500	2,886.896	1.000	2,500.750	5,000.500	7,500.250	10,000.000
CustomerId	10,000.000	15,690,940.569	71,936.186	15,565,701.000	15,628,528.250	15,690,738.000	15,753,233.750	15,815,690.000
CreditScore	10,000.000	650.529	96.653	350.000	584.000	652.000	718.000	850.000
Age	10,000.000	38.922	10.488	18.000	32.000	37.000	44.000	92.000
Tenure	10,000.000	5.013	2.892	0.000	3.000	5.000	7.000	10.000
Balance	10,000.000	76,485.889	62,397.405	0.000	0.000	97,198.540	127,644.240	250,898.090
NumOfProducts	10,000.000	1.530	0.582	1.000	1.000	1.000	2.000	4.000
HasCrCard	10,000.000	0.706	0.456	0.000	0.000	1.000	1.000	1.000
IsActiveMember	10,000.000	0.515	0.500	0.000	0.000	1.000	1.000	1.000
EstimatedSalary	10,000.000	100,090.240	57,510.493	11.580	51,002.110	100,193.915	149,388.247	199,992.480
Exited	10,000.000	0.204	0.403	0.000	0.000	0.000	0.000	1.000

# Create a profile report
profile = df.profile_report()                                     
profile

# Visualize features with pairplots
sns.pairplot(df,diag_kind='kde')

<seaborn.axisgrid.PairGrid at 0x242589bfbc8>

# Check target output for balance
df['Exited'].value_counts()

0    7963
1    2037
Name: Exited, dtype: int64

Observations:

• Raw dataset has 10000 records and 14 features
• Raw dataset has a mix of int64, float64, and object datatypes
• There are no missing values
• There are no duplicate rows
• Our dataset is imbalanced with only 25.6% of the customers as 'Exited'

Data Preparation:

• "RowNumber", "CustomerId", "Surname" features will be dropped as they do not add value to our model.
• "Exited" is our target output.
• "Gender" and "Geography" features will be converted from categorical to OneHot encoding.
• We will convert all remaining features to "float32".
• We will need to scale the dataset.

# Drop non-relevant features from dataset
df.drop(['RowNumber','CustomerId','Surname'], axis = 1, inplace = True)
df.head()

	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
0	619	France	Female	42	2	0.000	1	1	1	101,348.880	1
1	608	Spain	Female	41	1	83,807.860	1	0	1	112,542.580	0
2	502	France	Female	42	8	159,660.800	3	1	0	113,931.570	1
3	699	France	Female	39	1	0.000	2	0	0	93,826.630	0
4	850	Spain	Female	43	2	125,510.820	1	1	1	79,084.100	0

# Convert categorical features to OneHot Encoding
df = pd.get_dummies(df, drop_first=True)
df.head()

	CreditScore	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited	Geography_Germany	Geography_Spain	Gender_Male
0	619	42	2	0.000	1	1	1	101,348.880	1	0	0	0
1	608	41	1	83,807.860	1	0	1	112,542.580	0	0	1	0
2	502	42	8	159,660.800	3	1	0	113,931.570	1	0	0	0
3	699	39	1	0.000	2	0	0	93,826.630	0	0	0	0
4	850	43	2	125,510.820	1	1	1	79,084.100	0	0	1	0

# Convert all fields into the same data type - float
df = df.astype('float32')
df.head()

	CreditScore	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited	Geography_Germany	Geography_Spain	Gender_Male
0	619.000	42.000	2.000	0.000	1.000	1.000	1.000	101,348.883	1.000	0.000	0.000	0.000
1	608.000	41.000	1.000	83,807.859	1.000	0.000	1.000	112,542.578	0.000	0.000	1.000	0.000
2	502.000	42.000	8.000	159,660.797	3.000	1.000	0.000	113,931.570	1.000	0.000	0.000	0.000
3	699.000	39.000	1.000	0.000	2.000	0.000	0.000	93,826.633	0.000	0.000	0.000	0.000
4	850.000	43.000	2.000	125,510.820	1.000	1.000	1.000	79,084.102	0.000	0.000	1.000	0.000

# Display data types and determine if any missing values
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000 entries, 0 to 9999
Data columns (total 12 columns):
 #   Column             Non-Null Count  Dtype  
---  ------             --------------  -----  
 0   CreditScore        10000 non-null  float32
 1   Age                10000 non-null  float32
 2   Tenure             10000 non-null  float32
 3   Balance            10000 non-null  float32
 4   NumOfProducts      10000 non-null  float32
 5   HasCrCard          10000 non-null  float32
 6   IsActiveMember     10000 non-null  float32
 7   EstimatedSalary    10000 non-null  float32
 8   Exited             10000 non-null  float32
 9   Geography_Germany  10000 non-null  float32
 10  Geography_Spain    10000 non-null  float32
 11  Gender_Male        10000 non-null  float32
dtypes: float32(12)
memory usage: 468.9 KB

Split the dataset into train, validation, and test sets

# Separate predictors and the target field
X = df.drop(columns = 'Exited')
y = df['Exited']

# Split data into train, validation and test

# Split all data into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.20, random_state = 1)

# Further split train into train and validation sets
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size = 0.20, random_state = 1)

# Create copies of dataframes to avoid indexing warning
[X_train, X_val, X_test, y_train, y_val, y_test] = map(lambda x: x.copy(), [X_train, X_val, X_test, y_train, y_val, y_test])

# Scale / Normalize the dataset
scaler = StandardScaler()

X_train = scaler.fit_transform(X_train)
X_val = scaler.transform(X_val)
X_test = scaler.transform(X_test)

Create Helper Functon(s)

# Creating a function for evaluating model performance
def evaluate_nn(model, train, val, test):

    loss, acc_train = model.evaluate(train, y_train, verbose=0)
    loss, acc_val = model.evaluate(val, y_val, verbose=0)
    loss, acc_test = model.evaluate(test, y_test, verbose=0)
    
    return acc_train, acc_val, acc_test

Build Models

• We will build ANN models with ReLU, Sigmoid, and Tanh activation functions
• We will base our initial hidden layer node count on the average of our input and output layers: (11 + 1) / 2 = 6
• We will evaluate those models based on 'Accuracy' since this is the specific metric required by the problem statement

ReLU - SGD - 1 Hidden Layer

%%time
# Clear the session and set a random seed
tf.keras.backend.clear_session()
tf.random.set_seed(1)

# Initiate the model and add layers
nn_1 = tf.keras.models.Sequential([Input(shape=(X_train.shape[1],)),    # Input layer
                                   Dense(6, activation = 'relu'),       # Hidden layer with relu activation 
                                   Dense(1, activation = 'sigmoid')])   # Output layer

# Add loss function, optimizer and evaluation metric to the model
nn_1.compile(loss = 'binary_crossentropy', 
             optimizer = 'SGD',      # Use SGD optimizer initially
             metrics = ['accuracy']) # Evaluate model based on 'accuracy' per problem statement

# Train the model
nn_1.fit(X_train, y_train, validation_data = (X_val, y_val), epochs = 10, verbose = 1)

# Score train and test set and record the results
scores = evaluate_nn(nn_1, X_train, X_val, X_test)

# Store results in a dataframe
res = pd.DataFrame({'Method': ['NN - ReLU | SGD | 1HL'], 
                    'Accuracy Train': scores[0], 
                    'Accuracy Validation': scores[1],
                    'Accuracy Test': scores[2]})

res

Epoch 1/10
200/200 [==============================] - 1s 3ms/step - loss: 0.6649 - accuracy: 0.6438 - val_loss: 0.5614 - val_accuracy: 0.7694
Epoch 2/10
200/200 [==============================] - 0s 2ms/step - loss: 0.5300 - accuracy: 0.7947 - val_loss: 0.5137 - val_accuracy: 0.7931
Epoch 3/10
200/200 [==============================] - 0s 1ms/step - loss: 0.5028 - accuracy: 0.7980 - val_loss: 0.4955 - val_accuracy: 0.7931
Epoch 4/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4894 - accuracy: 0.7978 - val_loss: 0.4839 - val_accuracy: 0.7931
Epoch 5/10
200/200 [==============================] - 0s 945us/step - loss: 0.4798 - accuracy: 0.7980 - val_loss: 0.4753 - val_accuracy: 0.7931
Epoch 6/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4722 - accuracy: 0.7986 - val_loss: 0.4685 - val_accuracy: 0.7931
Epoch 7/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4659 - accuracy: 0.7981 - val_loss: 0.4628 - val_accuracy: 0.7944
Epoch 8/10
200/200 [==============================] - 0s 2ms/step - loss: 0.4605 - accuracy: 0.8006 - val_loss: 0.4578 - val_accuracy: 0.7956
Epoch 9/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4558 - accuracy: 0.8016 - val_loss: 0.4536 - val_accuracy: 0.7987
Epoch 10/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4517 - accuracy: 0.8020 - val_loss: 0.4500 - val_accuracy: 0.8000
Wall time: 4.22 s

	Method	Accuracy Train	Accuracy Validation	Accuracy Test
0	NN - ReLU | SGD | 1HL	0.803	0.800	0.802

Sigmoid - SGD - 1 Hidden Layer

%%time
# Clear the session and set a random seed
tf.keras.backend.clear_session()
tf.random.set_seed(1)

# Initiate the model and add layers
nn_2 = tf.keras.models.Sequential([Input(shape=(X_train.shape[1],)),    # Input layer
                                   Dense(6, activation = 'sigmoid'),    # Hidden layer with sigmoid activation    
                                   Dense(1, activation = 'sigmoid')])   # Output layer

# Add loss function, optimizer and evaluation metric to the model
nn_2.compile(loss = 'binary_crossentropy', 
             optimizer = 'SGD',      # Use SGD optimizer initially
             metrics = ['accuracy']) # Evaluate model based on 'accuracy' per problem statement

# Train the model
nn_2.fit(X_train, y_train, validation_data = (X_val, y_val), epochs = 10, verbose = 1)

# Score train and test set and record the results
scores = evaluate_nn(nn_2, X_train, X_val, X_test)

# Store results in a dataframe
res.loc[len(res)] = ['NN - Sigmoid | SGD | 1HL', scores[0], scores[1], scores[2]]
res

Epoch 1/10
200/200 [==============================] - 0s 1ms/step - loss: 0.6360 - accuracy: 0.6569 - val_loss: 0.5551 - val_accuracy: 0.7937
Epoch 2/10
200/200 [==============================] - 0s 1ms/step - loss: 0.5253 - accuracy: 0.7981 - val_loss: 0.5229 - val_accuracy: 0.7937
Epoch 3/10
200/200 [==============================] - 0s 1ms/step - loss: 0.5090 - accuracy: 0.7981 - val_loss: 0.5147 - val_accuracy: 0.7937
Epoch 4/10
200/200 [==============================] - 0s 1ms/step - loss: 0.5033 - accuracy: 0.7981 - val_loss: 0.5099 - val_accuracy: 0.7937
Epoch 5/10
200/200 [==============================] - 0s 965us/step - loss: 0.4994 - accuracy: 0.7981 - val_loss: 0.5059 - val_accuracy: 0.7937
Epoch 6/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4960 - accuracy: 0.7981 - val_loss: 0.5023 - val_accuracy: 0.7937
Epoch 7/10
200/200 [==============================] - 0s 950us/step - loss: 0.4927 - accuracy: 0.7981 - val_loss: 0.4988 - val_accuracy: 0.7937
Epoch 8/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4897 - accuracy: 0.7981 - val_loss: 0.4955 - val_accuracy: 0.7937
Epoch 9/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4867 - accuracy: 0.7981 - val_loss: 0.4923 - val_accuracy: 0.7937
Epoch 10/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4838 - accuracy: 0.7981 - val_loss: 0.4892 - val_accuracy: 0.7937
Wall time: 3.05 s

	Method	Accuracy Train	Accuracy Validation	Accuracy Test
0	NN - ReLU | SGD | 1HL	0.803	0.800	0.802
1	NN - Sigmoid | SGD | 1HL	0.798	0.794	0.793

Tanh - SGD - 1HL

%%time
# Clear the session and set a random seed
tf.keras.backend.clear_session()
tf.random.set_seed(1)

# Initiate the model and add layers
nn_3 = tf.keras.models.Sequential([Input(shape=(X_train.shape[1],)),    # Input layer
                                   Dense(6, activation = 'tanh'),       # Hidden layer with tanh activation 
                                   Dense(1, activation = 'sigmoid')])   # Output layer

# Add loss function, optimizer and evaluation metric to the model
nn_3.compile(loss = 'binary_crossentropy', 
             optimizer = 'SGD',      # Use SGD optimizer initially
             metrics = ['accuracy']) # Evaluate model based on 'accuracy' per problem statement

# Train the model
nn_3.fit(X_train, y_train, validation_data = (X_val, y_val), epochs = 10, verbose = 1)

# Score train and test set and record the results
scores = evaluate_nn(nn_3, X_train, X_val, X_test)

# Store results in a dataframe
res.loc[len(res)] = ['NN - Tanh | SGD | 1HL', scores[0], scores[1], scores[2]]
res

Epoch 1/10
  1/200 [..............................] - ETA: 0s - loss: 0.8141 - accuracy: 0.3438WARNING:tensorflow:Callbacks method `on_train_batch_end` is slow compared to the batch time (batch time: 0.0000s vs `on_train_batch_end` time: 0.0009s). Check your callbacks.
200/200 [==============================] - 0s 2ms/step - loss: 0.6484 - accuracy: 0.6175 - val_loss: 0.5844 - val_accuracy: 0.7419
Epoch 2/10
200/200 [==============================] - 0s 910us/step - loss: 0.5308 - accuracy: 0.7933 - val_loss: 0.5156 - val_accuracy: 0.7900
Epoch 3/10
200/200 [==============================] - 0s 950us/step - loss: 0.4875 - accuracy: 0.8008 - val_loss: 0.4859 - val_accuracy: 0.7937
Epoch 4/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4673 - accuracy: 0.7986 - val_loss: 0.4701 - val_accuracy: 0.7944
Epoch 5/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4557 - accuracy: 0.8000 - val_loss: 0.4604 - val_accuracy: 0.7981
Epoch 6/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4483 - accuracy: 0.8019 - val_loss: 0.4538 - val_accuracy: 0.7981
Epoch 7/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4431 - accuracy: 0.8020 - val_loss: 0.4491 - val_accuracy: 0.7962
Epoch 8/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4393 - accuracy: 0.8027 - val_loss: 0.4455 - val_accuracy: 0.7987
Epoch 9/10
200/200 [==============================] - 0s 945us/step - loss: 0.4365 - accuracy: 0.8048 - val_loss: 0.4428 - val_accuracy: 0.8000
Epoch 10/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4343 - accuracy: 0.8075 - val_loss: 0.4409 - val_accuracy: 0.7994
Wall time: 2.93 s

	Method	Accuracy Train	Accuracy Validation	Accuracy Test
0	NN - ReLU | SGD | 1HL	0.803	0.800	0.802
1	NN - Sigmoid | SGD | 1HL	0.798	0.794	0.793
2	NN - Tanh | SGD | 1HL	0.807	0.799	0.807

Model Improvements

• We identify the ReLU model as the slightly better performing model due better accuracy and lowest spread.
• We try improving its result by trying a few different optimizers
• We try improving its result by adding an additional hidden layer
• We wiil then try balancing the dataset using SMOTE and check again for any improvements

ReLU - Adam - 1 Hidden Layer

%%time
# Clear the session and set a random seed
tf.keras.backend.clear_session()
tf.random.set_seed(1)

# Initiate the model and add layers
nn_4 = tf.keras.models.Sequential([Input(shape=(X_train.shape[1],)),    # Input layer
                                   Dense(6, activation = 'relu'),       # Hidden layer with relu activation 
                                   Dense(1, activation = 'sigmoid')])   # Output layer

# Add loss function, optimizer and evaluation metric to the model
nn_4.compile(loss = 'binary_crossentropy', 
             optimizer = 'Adam',     # Try the Adam optimizer
             metrics = ['accuracy']) # Evaluate model based on 'accuracy' per problem statement

# Train the model
nn_4.fit(X_train, y_train, validation_data = (X_val, y_val), epochs = 10, verbose = 1)

# Score train and test set and record the results
scores = evaluate_nn(nn_4, X_train, X_val, X_test)

# Store results in a dataframe
res.loc[len(res)] = ['NN - ReLU | Adam | 1HL', scores[0], scores[1], scores[2]]
res

Epoch 1/10
200/200 [==============================] - 0s 1ms/step - loss: 0.7111 - accuracy: 0.5844 - val_loss: 0.5791 - val_accuracy: 0.7487
Epoch 2/10
200/200 [==============================] - 0s 980us/step - loss: 0.5177 - accuracy: 0.7870 - val_loss: 0.4843 - val_accuracy: 0.7962
Epoch 3/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4684 - accuracy: 0.8003 - val_loss: 0.4587 - val_accuracy: 0.7956
Epoch 4/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4522 - accuracy: 0.8020 - val_loss: 0.4478 - val_accuracy: 0.7969
Epoch 5/10
200/200 [==============================] - 0s 980us/step - loss: 0.4436 - accuracy: 0.8062 - val_loss: 0.4417 - val_accuracy: 0.8031
Epoch 6/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4376 - accuracy: 0.8095 - val_loss: 0.4376 - val_accuracy: 0.8044
Epoch 7/10
200/200 [==============================] - 0s 985us/step - loss: 0.4329 - accuracy: 0.8133 - val_loss: 0.4340 - val_accuracy: 0.8075
Epoch 8/10
200/200 [==============================] - 0s 990us/step - loss: 0.4287 - accuracy: 0.8138 - val_loss: 0.4311 - val_accuracy: 0.8119
Epoch 9/10
200/200 [==============================] - 0s 980us/step - loss: 0.4248 - accuracy: 0.8170 - val_loss: 0.4281 - val_accuracy: 0.8131
Epoch 10/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4211 - accuracy: 0.8209 - val_loss: 0.4262 - val_accuracy: 0.8156
Wall time: 2.88 s

	Method	Accuracy Train	Accuracy Validation	Accuracy Test
0	NN - ReLU | SGD | 1HL	0.803	0.800	0.802
1	NN - Sigmoid | SGD | 1HL	0.798	0.794	0.793
2	NN - Tanh | SGD | 1HL	0.807	0.799	0.807
3	NN - ReLU | Adam | 1HL	0.822	0.816	0.822

ReLU - RMSProp - 1 Hidden Layer

%%time
# Clear the session and set a random seed
tf.keras.backend.clear_session()
tf.random.set_seed(1)

# Initiate the model and add layers
nn_5 = tf.keras.models.Sequential([Input(shape=(X_train.shape[1],)),    # Input layer
                                   Dense(6, activation = 'relu'),       # Hidden layer with relu activation 
                                   Dense(1, activation = 'sigmoid')])   # Output layer

# Add loss function, optimizer and evaluation metric to the model
nn_5.compile(loss = 'binary_crossentropy', 
             optimizer = 'RMSprop',  # Try the RMSprop optimizer
             metrics = ['accuracy']) # Evaluate model based on 'accuracy' per problem statement

# Train the model
nn_5.fit(X_train, y_train, validation_data = (X_val, y_val), epochs = 10, verbose = 1)

# Score train and test set and record the results
scores = evaluate_nn(nn_5, X_train, X_val, X_test)

# Store results in a dataframe
res.loc[len(res)] = ['NN - ReLU | RMSProp | 1HL', scores[0], scores[1], scores[2]]
res

Epoch 1/10
200/200 [==============================] - 0s 1ms/step - loss: 0.6940 - accuracy: 0.6053 - val_loss: 0.5616 - val_accuracy: 0.7631
Epoch 2/10
200/200 [==============================] - 0s 1ms/step - loss: 0.5063 - accuracy: 0.7952 - val_loss: 0.4756 - val_accuracy: 0.7962
Epoch 3/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4653 - accuracy: 0.8027 - val_loss: 0.4550 - val_accuracy: 0.7975
Epoch 4/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4499 - accuracy: 0.8064 - val_loss: 0.4441 - val_accuracy: 0.8006
Epoch 5/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4409 - accuracy: 0.8106 - val_loss: 0.4381 - val_accuracy: 0.8031
Epoch 6/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4354 - accuracy: 0.8123 - val_loss: 0.4343 - val_accuracy: 0.8062
Epoch 7/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4311 - accuracy: 0.8167 - val_loss: 0.4318 - val_accuracy: 0.8131
Epoch 8/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4272 - accuracy: 0.8180 - val_loss: 0.4287 - val_accuracy: 0.8194
Epoch 9/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4236 - accuracy: 0.8211 - val_loss: 0.4266 - val_accuracy: 0.8194
Epoch 10/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4201 - accuracy: 0.8222 - val_loss: 0.4246 - val_accuracy: 0.8219
Wall time: 3.09 s

	Method	Accuracy Train	Accuracy Validation	Accuracy Test
0	NN - ReLU | SGD | 1HL	0.803	0.800	0.802
1	NN - Sigmoid | SGD | 1HL	0.798	0.794	0.793
2	NN - Tanh | SGD | 1HL	0.807	0.799	0.807
3	NN - ReLU | Adam | 1HL	0.822	0.816	0.822
4	NN - ReLU | RMSProp | 1HL	0.824	0.822	0.826

ReLU - RMSProp - 2 Hidden Layers

• We noted some improvemnts by using the RMSProp optimizer, so we will use this model to try for further improvements.

%%time
# Clear the session and set a random seed
tf.keras.backend.clear_session()
tf.random.set_seed(1)

# Initiate the model and add layers
nn_6 = tf.keras.models.Sequential([Input(shape=(X_train.shape[1],)),    # Input layer
                                   Dense(6, activation = 'relu'),       # Hidden layer with relu activation 
                                   Dense(6, activation = 'relu'),       # Add a second hidden layer
                                   Dense(1, activation = 'sigmoid')])   # Output layer

# Add loss function, optimizer and evaluation metric to the model
nn_6.compile(loss = 'binary_crossentropy', 
             optimizer = 'RMSprop',  # Try the RMSprop optimizer
             metrics = ['accuracy']) # Evaluate model based on 'accuracy' per problem statement

# Train the model
nn_6.fit(X_train, y_train, validation_data = (X_val, y_val), epochs = 10, verbose = 1)

# Score train and test set and record the results
scores = evaluate_nn(nn_6, X_train, X_val, X_test)

# Store results in a dataframe
res.loc[len(res)] = ['NN - ReLU | RMSProp | 2HL', scores[0], scores[1], scores[2]]
res

Epoch 1/10
200/200 [==============================] - 0s 1ms/step - loss: 0.5225 - accuracy: 0.7980 - val_loss: 0.4819 - val_accuracy: 0.7937
Epoch 2/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4730 - accuracy: 0.7981 - val_loss: 0.4567 - val_accuracy: 0.7944
Epoch 3/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4516 - accuracy: 0.8034 - val_loss: 0.4458 - val_accuracy: 0.8087
Epoch 4/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4410 - accuracy: 0.8083 - val_loss: 0.4406 - val_accuracy: 0.8112
Epoch 5/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4352 - accuracy: 0.8102 - val_loss: 0.4372 - val_accuracy: 0.8125
Epoch 6/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4313 - accuracy: 0.8097 - val_loss: 0.4349 - val_accuracy: 0.8150
Epoch 7/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4283 - accuracy: 0.8131 - val_loss: 0.4328 - val_accuracy: 0.8144
Epoch 8/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4254 - accuracy: 0.8142 - val_loss: 0.4304 - val_accuracy: 0.8106
Epoch 9/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4228 - accuracy: 0.8173 - val_loss: 0.4289 - val_accuracy: 0.8150
Epoch 10/10
200/200 [==============================] - 0s 1ms/step - loss: 0.4205 - accuracy: 0.8191 - val_loss: 0.4268 - val_accuracy: 0.8163
Wall time: 3.46 s

	Method	Accuracy Train	Accuracy Validation	Accuracy Test
0	NN - ReLU | SGD | 1HL	0.803	0.800	0.802
1	NN - Sigmoid | SGD | 1HL	0.798	0.794	0.793
2	NN - Tanh | SGD | 1HL	0.807	0.799	0.807
3	NN - ReLU | Adam | 1HL	0.822	0.816	0.822
4	NN - ReLU | RMSProp | 1HL	0.824	0.822	0.826
5	NN - ReLU | RMSProp | 2HL	0.822	0.816	0.821

ReLU - RMSprop - 1 Hidden Layer - SMOTE

• We note that there was not much improvement by adding the second hidden layer; we will revert to using ReLU - RMSprop - 1HL

# import SMOTE
from imblearn.over_sampling import SMOTE

# We apply SMOTE only on train data to not introduce any data leak
sm = SMOTE(random_state = 1)
X_train, y_train = sm.fit_resample(X_train, y_train)

%%time
# Clear the session and set a random seed
tf.keras.backend.clear_session()
tf.random.set_seed(1)

# Initiate the model and add layers
nn_6 = tf.keras.models.Sequential([Input(shape=(X_train.shape[1],)),    # Input layer
                                   Dense(6, activation = 'relu'),       # Hidden layer with relu activation
                                   Dense(1, activation = 'sigmoid')])   # Output layer

# Add loss function, optimizer and evaluation metric to the model
nn_6.compile(loss = 'binary_crossentropy', 
             optimizer = 'RMSprop',  # RMSprop Optimizer
             metrics = ['accuracy']) # Evaluate model based on 'accuracy' per problem statement

# Train the model
nn_6.fit(X_train, y_train, validation_data = (X_val, y_val), epochs = 10, verbose = 1)

# Score train and test set and record the results
scores = evaluate_nn(nn_6, X_train, X_val, X_test)

# Store results in a dataframe
res.loc[len(res)] = ['NN - reLU | RMSpop | 1HL | SMOTE', scores[0], scores[1], scores[2]]
res

Epoch 1/10
320/320 [==============================] - 1s 2ms/step - loss: 0.6780 - accuracy: 0.5860 - val_loss: 0.6473 - val_accuracy: 0.6481
Epoch 2/10
320/320 [==============================] - 0s 1ms/step - loss: 0.6031 - accuracy: 0.6763 - val_loss: 0.6113 - val_accuracy: 0.6837
Epoch 3/10
320/320 [==============================] - 0s 1ms/step - loss: 0.5823 - accuracy: 0.6976 - val_loss: 0.6012 - val_accuracy: 0.6988
Epoch 4/10
320/320 [==============================] - 0s 1ms/step - loss: 0.5713 - accuracy: 0.7056 - val_loss: 0.5851 - val_accuracy: 0.7050
Epoch 5/10
320/320 [==============================] - 0s 944us/step - loss: 0.5594 - accuracy: 0.7143 - val_loss: 0.5682 - val_accuracy: 0.7106
Epoch 6/10
320/320 [==============================] - 0s 1ms/step - loss: 0.5453 - accuracy: 0.7267 - val_loss: 0.5631 - val_accuracy: 0.7106
Epoch 7/10
320/320 [==============================] - 0s 1ms/step - loss: 0.5303 - accuracy: 0.7384 - val_loss: 0.5465 - val_accuracy: 0.7150
Epoch 8/10
320/320 [==============================] - 0s 972us/step - loss: 0.5147 - accuracy: 0.7489 - val_loss: 0.5405 - val_accuracy: 0.7163
Epoch 9/10
320/320 [==============================] - 0s 997us/step - loss: 0.5007 - accuracy: 0.7612 - val_loss: 0.5336 - val_accuracy: 0.7231
Epoch 10/10
320/320 [==============================] - 0s 966us/step - loss: 0.4898 - accuracy: 0.7713 - val_loss: 0.5341 - val_accuracy: 0.7188
Wall time: 4.85 s

	Method	Accuracy Train	Accuracy Validation	Accuracy Test
0	NN - ReLU | SGD | 1HL	0.803	0.800	0.802
1	NN - Sigmoid | SGD | 1HL	0.798	0.794	0.793
2	NN - Tanh | SGD | 1HL	0.807	0.799	0.807
3	NN - ReLU | Adam | 1HL	0.822	0.816	0.822
4	NN - ReLU | RMSProp | 1HL	0.824	0.822	0.826
5	NN - ReLU | RMSProp | 2HL	0.822	0.816	0.821
6	NN - reLU | RMSpop | 1HL | SMOTE	0.773	0.719	0.754

Conclusion

# Display all model results
res

	Method	Accuracy Train	Accuracy Validation	Accuracy Test
0	NN - ReLU | SGD | 1HL	0.803	0.800	0.802
1	NN - Sigmoid | SGD | 1HL	0.798	0.794	0.793
2	NN - Tanh | SGD | 1HL	0.807	0.799	0.807
3	NN - ReLU | Adam | 1HL	0.822	0.816	0.822
4	NN - ReLU | RMSProp | 1HL	0.824	0.822	0.826
5	NN - ReLU | RMSProp | 2HL	0.822	0.816	0.821
6	NN - reLU | RMSpop | 1HL | SMOTE	0.773	0.719	0.754

Remarks:

• Based on the accuracy metric and lower-spread, the ReLU model with RMSProp optimizer and 1 Hidden Layer appears to be the best performer of models tested.
• We may be able to improve the model prediction of 'Exited' by using a different metric depending on the bank's overall objectives.

# Display results for best performer
res.loc[4]

Method                 NN - ReLU | RMSProp | 1HL
Accuracy Train                             0.824
Accuracy Validation                        0.822
Accuracy Test                              0.826
Name: 4, dtype: object

# Score test set with the final model and compare to actuals
p = nn_5.predict(X_test).flatten()
pd.crosstab(y_test, np.where(p > 0.5, 1,0), colnames = ['Predictions'])

Predictions	0	1
Exited
0.000	1530	55
1.000	292	123